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Abstract 

We introduce a new approach of computing the automorphism group 
and the field of moduli of points p = [C] in the moduli space of hyperel- 
liptic curves Ti g . Further, we show that for every moduli point p £ H g (L) 
such that the reduced automorphism group of p has at least two involu- 
tions, there exists a representative C of the isomorphism class p which is 
defined over L. 

1 Introduction 

The purpose of this note is to introduce some new techniques of computing the 
automorphism group and the field of moduli of genus g hyperelliptic curves. 
Former results by many authors have focused on hyperelliptic curves of small 
genus, see [8], [3], [7], ^U]j et. al. We aim to find a method which would 
work for any genus. 

Let C denote a genus g hyperelliptic curve defined over an algebraically 
closed field k of characteristic zero and G := Aut{C) its automorphism group. 
We denote by TL g the moduli space of genus g hyperelliptic curves and by C g the 
locus in Tig of hyperelliptic curves with extra involutions. C g is a (^-dimensional 
rational variety, see [5]. Equation [21 gives a normal form for curves in C g . This 
normal form depends on parameters a±, . . . ,a g € k, such that the discriminant 
of the right side A(a l5 . . . , a g ) =/= 0. Dihedral invariants (iti, . . . , u g ) were intro- 
duced by Gutierrez and this author in [S]. The tuples u = (ui, . . . , u g ) (such that 
A u =/= 0) are in one-to-one correspondence with isomorphism classes of genus g 
hyperelliptic curves with automorphism group the Klein 4-group. Thus, dihe- 
dral invariants Ui,...,u g yield a birational parameterization of the locus C g . 
Computationally these invariants give an efficient way of determining a generic 
point of the moduli space C g . Normally, this is accomplished by invariants of 
GL2(k) acting on the space of binary forms of degree 2g + 2. These GL2(k)- 
invariants are not known for g > 3. However, dihedral invariants are explicitly 
defined for all genera. 

The full automorphism groups of hyperelliptic curves are determined in [2] 
and pp. Most of these groups have non- hyperelliptic involutions (i.e., the cor- 



responding curve is in C g ). For each group G that occurs as full automorphism 
group of genus g curves one determines the G-locus in C g in terms of the dihe- 
dral invariants. Given a genus g curve C we first determine if C G C g . Then we 
compute its dihedral invariants and determine the locus Cg that they satisfy. 
This determines Aut(C). Present algorithms of computing the automorphism 
group of a hyperelliptic curve Y 2 = F(X) are based on computing the roots 
of F(X) and then finding fractional linear transformations that permute these 
roots. The algorithm we propose requires only determining the normal form of 
C (i.e., Eq.[2J. This requires solving a system of <?-equations and four unknowns. 
For curves which have at least two involutions in their reduced automorphism 
group we find a nice condition on the dihedral invariants. 

For C ^ C g similar methods can be used. If |Aitt(C)| > 2 and C g" C g , then 
C has an automorphism of order TV, where TV is as in LemmaOH For small genus 
these curves can be classified by ad-hoc methods. In general one needs to find 
invariants of such spaces for all TV > 2 and implement similar methods as above. 
We intend this as the object of further research. 

In section 4, wc introduce how to compute the field of moduli of genus 
g hyperelliptic curves with automorphism group of order > 4. Let M g (rcsp., 
Ti g ) be the moduli space of algebraic curves (resp., hyperelliptic curves) of genus 
g defined over k and L a subfield of k. It is well known that M. g (resp., Ti g ) 
is a 3g — 3 (resp., 2g — 1) dimensional variety. If C is a genus g curve defined 
over L, then clearly [C] G Ai g (L). However, the converse is not true. In other 
words, the moduli space M g of algebraic curves of genus g is a coarse moduli 
space. The answer is not obvious if we restrict ourselves to the singular points 
of M g . Singular points of M g (resp., H g ) correspond to isomorphism classes 
of curves with nontrivial automorphism groups (resp., automorphism groups of 
order > 2 ). In general, we conjecture that for a singular point p G M g (L) 
(resp., p G Hg(Lj) there is always a curve C defined over L which correspond 
to p. We focus on Tig. A point p = [C] G H g is given by the g-tuple of dihedral 
invariants. We denote by Aut{p) the automorphism group of any representative 
C of p. More precisely, for hyperelliptic curves we conjecture the following: 

Conjecture 1: Let p G H g (L) such that \Aut(p)\ > 2. There exists a represen- 
tative C of the isomorphism class p which is defined over L. 

In this paper we show how dihedral invariants can be used to prove some special 
cases of this conjecture. A detailed discussion on this problem is intended in 
The condition |Aut(p)| > 2 of the above conjecture can not be dropped. 
Determining exactly the points p G H g \ C g where such rational model C does 
not exist is still an open problem. For g = 2 Mestrc (1991) found an algorithm 
which determines such points. It is based on classical invariants of binary sextics. 

Notation: Throughout this paper k denotes an algebraically closed field of 
characteristic zero, g an integer > 2, and C a hyperelliptic curve of genus g. 
Ai g (resp., Tig) is the moduli space of curves (resp., hyperelliptic curves) defined 
over k. Further, V4 denotes the Klein 4-group and D^ n (resp., Z n ) the dihedral 
group of order 2n (resp., cyclic group of order n). 
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2 Dihedral invariants of hyperelliptic curves 



Let k be an algebraically closed field of characteristic zero and G be a genus g 
hyperelliptic curve given by the equation Y 2 = F(X), where deg(F) = 2g + 2. 
Denote the function field of G by K := k(X,Y). Then, k(X) is the unique 
degree 2 genus zero subfield of K. We identify the places of k(X) with the 
points of P 1 = k U {oo} in the natural way (the place X = a gets identified 
with the point a <G P 1 ). Then, K is a quadratic extension field of k(X) ramified 
exactly at n = 2g + 2 places a\, . . . ,a n of k{X). The corresponding places 
of K are called the Weierstrass points of K. Let V := {cti, ■ • ■ ,ct n }- Thus, 
K = k(X,Y), where 

y 2 = [] (*-«)• (!) 

Let G = Aut(K/k). Since is the only genus subfield of degree 2 of K, 
then G fixes k(X). Thus, Go := Gal(K/k(X)) = (zq), with Zq = 1, is central in 
G. We call the reduced automorphism group of K the group G := G/Go- 
Then, G is naturally isomorphic to the subgroup of Aut{k{X)/k) induced by 
G. We have a natural isomorphism V := PGLi{k) — > Aut(k(X)/k). The action 
of r on the places of k{X) corresponds under the above identification to the 
usual action on P 1 by fractional linear transformations t i— > ^rg. Further, G 
permutes on, . . . ,a n . This yields an embedding G ^ S n . 

Because K is the unique degree 2 extension of k(X) ramified exactly at 
o;i, ■ ■ ■ i oi n , each automorphism of k{X) permuting these n places extends to 
an automorphism of K. Thus, G is the stabilizer in Aut(k(X)/k) of the set 
V. Hence under the isomorphism T i— > Aut(k(X)/k) 1 G corresponds to the 
stabilizer T-p in T of the n-set V. 

An extra involution of K is an involution in G which is different from zo (the 
hyperelliptic involution). If z\ is an extra involution and zq the hyperelliptic 
one, then Z2 '■= zq z\ is another extra involution. So the extra involutions come 
naturally in pairs. Suppose z\ is an extra involution of K. Let z-i := z\ zq, 
where zq is the hyperelliptic involution. Then K = k(X,Y) with equation 

Y 2 = X 2a+2 + a g X 2a + ■■■ + a x X 2 + 1 (2) 

see 0. The dihedral group H := D2 g +2 = (Ti)T2) acts on k(a±,...,a g ) as 
follows: 

Ti : a. t — > e 2l a 4 , /or i = 1, . . . , g 

r -i r-9 + 1 i 
r 2 : aj — > a s+ i_ i; /or »=!,...,[— —J 

The fixed field k(ai, . . . , a g ) H is the same as the function field of the variety C g . 
The invariants of such action are 

m := af~ ?+1 dj + a^' +1 a ff _ i+ i, /or 1 < i < g (3) 
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and are called dihedral invariants for the genus g and the tuple 

u := (tii, ■■■■ > Ug) 

is called the tuple of dihedral invariants, see jS] for details. 

It is easily seen that u = if and only if a\ = a g = 0. In this case re- 
placing eti,a g by a,2,a g -i in the formula above would give new invariants. In 
jS] it is shown that k(C g ) = k(m, . . . ,u g ). The (2g + 2)-degree field extension 
fc(eti, . . . , a g )/k(ui, . . . , u g ) has equation 

2 9+l a 2g+2 _ 2 g+l ^ Q g+1 + ^+1 = q (4) 

and the map 

$: fc\{A^0} -^C g 

(ai,...,a g ) -* (ui, . . . ,u g ) 

has Jacobian zero exactly on points which correspond to curves C € C g such 
that V 4 ^ G. 

3 Automorphism groups 

In this section we suggest an algorithm for computing the full automorphism 
group of hyperclliptic curves. Let C be a genus g hypcrclliptic curve with 
equation Y 2 = F(X) where dcg(F) = 2g + 2. Existing algorithms are based on 
finding all automorphisms of C. Instead, we search for only one automorphism 
(non-hypcrelliptic) of C of order N. Most of the time N = 2 is enough since the 
majority of groups of order > 2 that occur as full automorphism groups have 
non-hypcrelliptic involutions. It is well known that the order of a non-trivial 
automorphism of a hyperclliptic curve is 2 < N < 2(2g + 1), where 2(2_g + 1) is 
known as the Wiman's bound. 

If an automorphism of order N = 2 exists then C € C g and we use dihedral 
invariants to determine the automorphism group. We illustrate with curves of 
small genus. 

The case g = 2 has been studied in ^21- Every point in M.i is a triple 
(ii j 225*3) °f absolute invariants. We state the results of ^2 without proofs. 

Lemma 1. Let C be a genus 2 curve such that G := Aut(C) has an extra 
involution and u = (1*1,1*2) its dihedral invariants. Then, 

a) G = Z3 xDg if and only if (1*1,^2) = (0, 0) or [u\, u%) = (6750, 450). 

b) G = GL 2 (3) if and only if(ui,u 2 ) = (-250,50). 

c) G^ D 12 if and only if u\ - 220w 2 - 16«i + 4500 = 0, for u 2 ^ 18, 140 + 
60^, 50. 

d) G^D S if and only if 2u\ - u\ = 0, for u 2 + 2,18,0,50,450. Cases 
u 2 = 0, 450 and u = 50 are reduced to cases a) and b) respectively. 
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The mapping <I> : (1*1,1*2) — ► (11,12,13), gives a birational parameterization 
of £2- The fibers of 4> of cardinality > 1 correspond to those curves C with 
|.i4.uf(C)| > 4. Dihedral invariants U\,U2 are given explicitly as rational func- 
tions of 11, 42,23. The curve Y 2 = X 6 — X is the only genus 2 curve (up to 
isomorphism) which has extra automorphisms and is not in £2- The automor- 
phism group in this case is Z 10 , see Thus, if C G £2 we determine Aut(C) 
via Lemma 3.1., otherwise C is isomorphic to Y 2 = X 6 — X or AutiC) = 2,2- 

The case g = 3 is given as an application in [5]. Let C G £3 with equation 
as in Eq. [21 Dihedral invariants are u\ = af + af, «2 = («i + 03)025 U3 = 2aia3. 
The analogue of Lemma 3.1 is proved in |Hj for g = 3. 

This technique can be used successfully for all g. We have implemented 
programs that determine Aut{C) for C £ C g and for g = 2, 3, 4, 5, 6. In order to 
compute the automorphism group of a curve C € C g we transform this curve to 
its normal form (i.e., Eq. [2J and then compute its dihedral invariants. If these 
invariants satisfy any locus Cg then the automorphism group is G, otherwise the 
automorphism group is V4. The following lemma determines a nice condition 
for G to have at least two involutions. 

Lemma 2. For a curve C £ C g the reduced automorphism group has at least 
two involutions if and only if 

{2^- 1 ul + u g +1 ){2<'- 1 ul-u g +1 ) = Q (5) 

Proof. Let C £ C g . Then, there is an involution z\ £ G which fixes no Weicr- 
strass points of C, see the proof of lemma 1 in Thus, z\(X) = —X. Let 
Z2 7^ z\ be another involution in G. Since, Z2 7^ Z\ then zs(X) = ^, where 
m 2 = 1. Then, V4 = (zx, z 2 ) » G and z 2 or Z\ Z2 is the transformation X — > A, 
say 22 PO = X' ^ -9 ^ s oc ^ we have = {±ai,±— , . . . , ±a„,±— }, where 
n = [^-], otherwise V contains also two points ±P. Thus, ±P can be either 
fixed or permuted by Zi{X) = y- Hence, they are ±1 or ±1, where I 2 = 1. 
The equation of C is given by 

n 

Y 2 = ]J(X 4 -A 4 X 2 + 1), if g is odd 

i=i 

n 

Y 2 = {X 2 ±l)\\{X i - \ t X 2 + 1), if g is even. 
i=i 

Let s := Xi + ■ ■ ■ + X n . If g is odd then a\ = a g = — s. Then, mi = 2s 9+1 
and u g = 2s 2 and they satisfy Eq. [3J If Z2(X) = fixes two points of V 
then one of the factors of the equation is X 4 — 1. Then, a\ = (—1)3+1 s anc ] 
a g = (-1)3TT s . Hence, a g +1 = a g +1 = -s 9+1 and u x = -2s 9+1 , u g = -2s 2 . 
Then, 23~ 1 u 2 1 +u g +1 = 0. 

If g is even and {±1} C V then a\ = a g = s + 1. If {±7} C V then 
ai = a g = 1 — s. In both cases 2 9 ~ 1 u 2 — = 0. The converse goes similarly. 

□ 
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Remark 1. If 2 9 ~ 1 u 2 + u 9+1 = 0, then one of the involutions z 2 , z\z 2 of G 
lifts to an element of order 4 in G. If 2 B u\ — u B g +1 = both of them lift to 
involutions in G. 

For C C g we check if C has automorphisms of order 3 < TV < 2(2g + 1), 
see Wiman . The following lemma is a consequence of [2] and gives possible 
values for TV. We only sketch the proof. 

Lemma 3. Let C be a genus g hyperelliptic curve with an automorphism of 
order TV > 2. Then either TV = 3, 4 or one of the following holds; 

i) TV|(2 5 + 1) orTV|2.g and TV < g (then Aut(C) ^ Z 2N ) 

ii) N\2g and TV is an even number such that 6 < TV < 2g — 2. 
Hi) TV = 4TV' such that N'\g and TV' < g. 

Proof. Let C be a genus g hyperelliptic curve with extra automorphisms such 
that C ^ C g . Then, the automorphism group of C is isomorphic to one of 
the following: SX 2 (3), SX 2 (5), W 3 , H N / 2 , U N / 2 , G N / 2 , % 2 n where TV | 2g + 1 
or TV | 2g and TV < g; see [2] for definitions of these groups. All other groups 
listed in Table 2 in [5] contain at least two involutions, hence they correspond 
to curves in C g . The only groups in the above list that might not contain an 
element of order 2, 3, or 4 are U^/ 2 , Gj^/ 2 . The group Gjy/2 (resp., Upj/ 2 ) has 
an element of order TV where TV is as above. 

□ 

To have a complete algorithm that works for any g > 2, one needs to classify 
(up to isomorphism) curves of genus g which are not in the locus C g . In order 
to do this, we need invariants which classify isomorphism classes of curves with 
an automorphism of order TV > 2. However, for small genus ad-hoc methods 
can be used to identify such groups. 

4 Field of moduli 

In this section we introduce a method to compute the field of moduli of hy- 
perelliptic curves with extra automorphisms. Until recently this was an open 
problem even for g = 2. Further, we state some open questions for higher genus 
and prove Conjecture 1 for p G TL g such that the reduced automorphism group 
of p has at least two involutions. 

Let C be a genus g hyperelliptic curve defined over k. We can write the 
equation of C as follows 

Y 2 = X(X - lXX 2 -"- 1 + c 2g - 2 X 29 - 2 + ■ ■ ■ + Cl X + c ) 

where the discriminant A of the right side is nonzero. Then, there is a map 

$i : fc 2 ^ 1 \ {A ji 0} - H g 
(co,...,c 29 - 2 ) ->p = [C] 
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of degree d = Ag{g + l)(2g + 1). We denote by J$ the Jacobian matrix of a map 
Then Conjecture 1 can be stated as follows: 

Conjecture 2: For each p in the locus det(J$ 1 ) = such that p £ Ji g (L) there 
exists a representative C of the isomorphism class p which is defined over L. 

For g = 2 this conjecture is a theorem as shown in PJ. The main result in 
P] is to prove the case when automorphism group is V4. A method of Mestre 
is generalized which uses covariants of order 2 of binary sextics and a result 
of Clebsch. Such a method probably could be generalized to higher genus as 
claimed by Mestre |S] and Weber |14j . 

Remark 2. There is a mistake in the proof of Theorem 2 in J^. In other words, 
the proof is incorrect when the Clebsch invariant C10 = 0. However, it can easily 
be fixed. A correct version of the algorithm has been implemented in Magma by 
P. van Wamelen. 

For g = 3 the conjecture is proven by Gutierrez and this author for all points 
p with |A«t(p)| > 4, see The proof uses dihedral invariants of hypcrclliptic 
curves. A generalization of the method used in jS], ^1] for p £ H3 such that 
Aut(p) = V4 would complete the case g — 3. 

Next we focus on the locus C g . Let C £ C g . Then, C can be written in the 
normal form as in equation [3 The map 

(at, ...,a g )-> (ui, ...,%) 

has degree d = 2g + 2. We ask a similar question as in Conjecture 2. Let p be 
in the locus det(J$ 1 ) = such that p £ H g (L). Is there a representative C of 
the isomorphism class p which is defined over LI 
The determinant of the Jacobian matrix is 

det(Ji) = {23- 1 u\ + ul +1 ) {23- 1 u\ - u a g +1 ). 

The locus det(J$) = corresponds exactly to the hyperelliptic curves with 
V4 G as shown by Lemma 3.2. 

Theorem 1. For each p in the locus det(J,j>) = such that p £ Tt g (L) there 
exists a representative C of the isomorphism class p which is defined over L. 
Moreover, the equation of C over L is given by 

C : Y 2 = uiX 29+2 + Ul X 2g + U2X 29 - 2 + ---±u g X 2 + 2, (6) 

where the coefficient of X 2 is u g (resp., —U g ) when 2 9 ~ l u\ — u 9 g +l = (resp., 
2 9 - 1 u 2 + u 9+1 = 0). 

Proof. Let p = (iti, . . . ,u g ) £ C g (L) such that 2 9 ~ 1 u\ - u 9+1 = 0. All we need 
to show is that the dihedral invariants of C satisfy the locus det(J$) = 0. By 
the appropriate transformation C can be written as 

Y 2 = X 29 + 2 + (^)i*r . X 29 + Y ^±i^ ■ ■ X 2 * + 1. 

2—1 
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Then, its dihedral invariants are 



m\ Ul i t u 9\g+i t Ul \Q 29 lu l + u9 g +1 

MC) = T + i t ] ( T } wZT~' ^ c ) = u - 

Substituting = 2 g ~ Y u\ we get tii(C) = u\. Thus, C is in the isomorphism 
class determined by p and defined over L. 

Let p = (tii, ■ ■ ■ ! u g ) G C g {L) such that 2 g ~ 1 u\ + u 9 g +1 = 0. This case occurs 
only when g is odd, see the proof of Lemma 3.2. We transform C as above 
and have tti(C) = u\ and u g (C) = — u g . They are the other tuple (u\, — u g ) 
which correspond to p. This completes the proof. 

□ 

The following is a consequence of Lemma 3.2. and Theorem 4.1. 

Corollary 1. Conjecture 1 holds for all p € C g such that the reduced automor- 
phism group of p has at least two involutions. 



5 Closing remarks 

Conjecture 1 was stated for the first time during a talk of the author in ANTS 
V, see |3 ■ It can be generalized to M g instead of H g . However, little is known 
about the loci Mg (i-e., locus of curves in Mg with full automorphism group 
G). In we introduce an algorithm that would classify such groups G for 
all g and give a complete list of "large" groups for g < 10. However, finding 
invariants that classify curves with automorphism group G is not an easy task, 
since the equations describing non-hyperelliptic curves are more complicated 
then the hyperelliptic case. A more theoretical approach on singular points of 
M g probably would produce better results on Conjecture 1. At this time we 
are not aware of any such results. 

Our approach would work (with necessary adjustments) even in positive 
characteristic. However, the goal of this note was to introduce such method 
rather than explore it to the full extent. 

Computationally, dihedral invariants give an efficient way of determining a 
point of the moduli space C g . Using such invariants in positive characteris- 
tic could have applications in the arithmetic of hyperelliptic curves, including 
cryptography. 
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